-
Notifications
You must be signed in to change notification settings - Fork 14
feat: Add .map() Method to Flow DSL for Map Step Support
#218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add .map() Method to Flow DSL for Map Step Support
#218
Conversation
🦋 Changeset detectedLatest commit: d52a0e0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 7 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
.map() Method to Flow DSL for Map Step Support
07f0d62 to
bb3491c
Compare
|
View your CI Pipeline Execution ↗ for commit d52a0e0
☁️ Nx Cloud last updated this comment at |
bb3491c to
429b748
Compare
b77f743 to
2b1f97e
Compare
429b748 to
0ed394f
Compare
2b1f97e to
fcb8222
Compare
252a37d to
83609aa
Compare
fcb8222 to
7422f2a
Compare
8e2d37b to
18fd9d1
Compare
18fd9d1 to
53cc331
Compare
7422f2a to
c1c4c34
Compare
53cc331 to
43c4799
Compare
c1c4c34 to
8ec0268
Compare
43c4799 to
26104e7
Compare
…nts, and edge cases - Introduced new integration tests for flow compilation involving map steps - Added runtime validation tests for step dependencies and slug validation - Included type inference validation tests for map method constraints - Covered edge cases such as empty arrays and various runtime options - Added tests for flow with only map steps and multiple independent chains - Ensured correct parameter ordering and handling of dependencies in compiled SQL - Expanded coverage for root and dependent map compilation scenarios - Included tests for flow with only map steps and chaining behaviors - Improved test suite robustness for map step handling in flow compilation
26104e7 to
d52a0e0
Compare
8ec0268 to
02056ce
Compare
🔍 Preview Deployment: Website✅ Deployment successful! 🔗 Preview URL: https://pr-218.pgflow.pages.dev 📝 Details:
_Last updated: _ |
🔍 Preview Deployment: Playground✅ Deployment successful! 🔗 Preview URL: https://pr-218--pgflow-demo.netlify.app 📝 Details:
_Last updated: _ |
Merge activity
|
## Overview
This PR introduces the `.map()` method to the Flow DSL, enabling users to define map-type steps that process arrays element-by-element. This completes the DSL support for the map step infrastructure that was previously added to the SQL Core layer.
## Problem
Previously, while the SQL Core supported map steps (with `step_type => 'map'`), there was no way to define them via the TypeScript DSL. Users had to manually write SQL or work around this limitation.
## Solution
The new `.map()` method provides a type-safe, intuitive way to define map steps with these key characteristics:
### API Design
```
// Root map - processes flow input array
new Flow<string[]>({ slug: 'text_processing' })
.map({ slug: 'normalize' }, (text) => text.trim().toLowerCase())
// Dependent map - processes another step's array output
new Flow<{}>({ slug: 'workflow' })
.array({ slug: 'fetch_items' }, () => fetchData())
.map({ slug: 'enrich', array: 'fetch_items' }, (item) => ({
...item,
enriched: true
}))
```
### Key Features
1. **Intuitive dependency syntax**
- Omit `array` property for root maps (processes flow input)
- Use `array: 'stepSlug'` for dependent maps (single dependency only)
2. **Different handler signature**
- Map handler: `(item, context) => Json`
- Regular step handler: `(input, context) => Json`
- Map handlers receive individual array elements, not the full input object
3. **Type safety**
- Enforces Json-compatible types for inputs and outputs
- Full TypeScript inference for item types based on dependencies
- Compile-time validation that dependencies return arrays
4. **SQL generation**
- Correctly adds `step_type => 'map'` parameter to `pgflow.add_step()`
- Maintains proper parameter ordering: dependencies, options, then step_type
## Implementation Details
### Changes Made
1. **Extended StepDefinition interface** (`dsl.ts`)
- Added optional `stepType?: 'single' | 'map'` property
- Defaults to 'single' for backward compatibility
2. **Implemented** **`.map()`** **method** (`dsl.ts`)
- Complex type signatures for proper inference
- Runtime validation for dependencies
- Stores `stepType: 'map'` in step definition
3. **Updated compile function** (`compile-flow.ts`)
- Checks for `stepType` property
- Adds `step_type => 'map'` parameter when needed
4. **Comprehensive test coverage**
- Type tests: 20 tests covering constraints and inference
- Runtime tests: 12 tests for SQL generation
- Integration tests: 12 tests for complete workflows
- Example flows demonstrating various use cases
### Technical Decisions
- **Optional** **`array`** **key**: Mirrors the `dependsOn` pattern (no key = root step)
- **Single dependency constraint**: Enforced at both type and runtime levels
- **Handler signature difference**: Clear distinction between map and regular steps
- **Type parameter ordering**: Worked around TypeScript constraints for optional parameters
## Testing
The implementation follows TDD principles with:
- ✅ Type safety tests (inference, constraints, edge cases)
- ✅ SQL generation tests (parameter ordering, step_type inclusion)
- ✅ Integration tests (complete workflows, validation)
- ✅ Example flows (data processing, ETL, parallel maps)
## Migration Guide
No breaking changes. Existing flows continue to work unchanged. To use map steps:
1. For root maps (array input):
2. For dependent maps:
## Future Considerations
- The `input.run` convention will be removed in future versions
- Map steps are ready for this transition (handlers don't use `input.run`)
- Type system properly handles both root and dependent map cases
## Checklist
- [x] Implementation complete with type safety
- [x] SQL generation working correctly
- [x] Comprehensive test coverage
- [x] Example flows created
- [x] No breaking changes
- [x] Documentation in changeset
## Related Issues
Part of the map infrastructure implementation series (see [PLAN.md](http://PLAN.md))

Overview
This PR introduces the
.map()method to the Flow DSL, enabling users to define map-type steps that process arrays element-by-element. This completes the DSL support for the map step infrastructure that was previously added to the SQL Core layer.Problem
Previously, while the SQL Core supported map steps (with
step_type => 'map'), there was no way to define them via the TypeScript DSL. Users had to manually write SQL or work around this limitation.Solution
The new
.map()method provides a type-safe, intuitive way to define map steps with these key characteristics:API Design
Key Features
arrayproperty for root maps (processes flow input)array: 'stepSlug'for dependent maps (single dependency only)(item, context) => Json(input, context) => Jsonstep_type => 'map'parameter topgflow.add_step()Implementation Details
Changes Made
dsl.ts)stepType?: 'single' | 'map'property.map()method (dsl.ts)stepType: 'map'in step definitioncompile-flow.ts)stepTypepropertystep_type => 'map'parameter when neededTechnical Decisions
arraykey: Mirrors thedependsOnpattern (no key = root step)Testing
The implementation follows TDD principles with:
Migration Guide
No breaking changes. Existing flows continue to work unchanged. To use map steps:
Future Considerations
input.runconvention will be removed in future versionsinput.run)Checklist
Related Issues
Part of the map infrastructure implementation series (see PLAN.md)